Papers with random and diversity-driven sampling
A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT (2026.findings-acl)
Copied to clipboard
| Challenge: | Diversity has been gaining interest in the NLP community in recent years. |
| Approach: | They propose to use diversity-driven sampling to pre-train models on French with a fixed compute budget. |
| Outcome: | The diversity-driven sampling reduces the pre-training dataset by 94% and the pretraining time by 73% while maintaining comparable performance. |